feat: add Vietnamese language support#1013
Merged
thatjuan merged 3 commits intoJun 26, 2026
Merged
Conversation
Vietnamese is an isolating (analytic) language where words do not inflect, so the stemmer is an identity function. Adds: - Vietnamese stemmer (identity function) - Vietnamese stopwords list - Vietnamese splitter regex with full diacritics support - Tokenizer test for Vietnamese
Vietnamese diacritics encode distinct vowels and tones, so folding them to ASCII changes word meaning. normalizeToken applied replaceDiacritics to every language, which partially stripped Vietnamese tokens (e.g. "tài" -> "tai", "trình" -> "trinh") while leaving Latin Extended Additional characters intact, producing inconsistent, lossy tokens. Skip replaceDiacritics for languages whose diacritics are significant, tracked by a new LANGUAGES_WITH_SIGNIFICANT_DIACRITICS set (currently Vietnamese). This fixes the new Vietnamese tokenizer test, which expects full diacritic preservation.
…exports - Drop dead underscore-joined compound stopwords (có_thể, một_cách, ngay_cả): the tokenizer splits on whitespace, so these never match real input, and their head words (có, một, ngay) are already individual stopwords. - Commit the generated @orama/stemmers/vietnamese and @orama/stopwords/vietnamese package.json exports (produced by `pnpm build`), matching the existing sanskrit convention so the published packages expose the subpaths.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Vietnamese (vi) language support to Orama, including:
packages/stemmers/lib/vi.js): Identity function — Vietnamese is an isolating (analytic) language where words do not inflect, so no morphological stemming is neededpackages/stopwords/lib/vi.js): Common Vietnamese function words (conjunctions, prepositions, particles, pronouns, etc.)languages.ts: Covers all Vietnamese diacritics (ă, â, ê, ô, ơ, ư, đ and all tone marks)stemmersandstopwordsbuild scriptsWhy Vietnamese?
Vietnamese is spoken by ~100 million native speakers and is one of the most widely used languages in Southeast Asia. Adding support enables Orama users to build search for Vietnamese content.
Technical Notes
Vietnamese is an isolating language — unlike European languages, words do not change form (no conjugation, declension, or inflection). This means:
return word)Test plan
pnpm testpasses for the tokenizer tests